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Traditional research literature distinguishes bétween 


_ two general types of threats to the generalizability of experinental ‘ 


findings: Internal validity and external validity. Relatively minor 
ia@portance has been attached to external validity in educational - 
literature.’ Bracht and Glass elaborate on external validity and deal ve 
with two types: Population validity and ecologicaél validity. Snow 
indicates that the biggest threat to external validity occurs when 

the experiment does not fit the phenomena being studied. Traditional 


‘experiments are nonecologically oriented, tend to manipulate the ° 


subject for experiaental convenience, and yield results which are _ 
non-representative. Historically, basic research has emphasized 

internal validity and applied research has been concerned with 

immediate problem solving (external generalizability). - . 
Generalizability must be the sole factor in judging all good ° ' 
research, and the fundamental design principle governing . . 
generalizability is representativeness. There are a number of 

strategies educational researchers may use to build significant and 


“generalizable studies, for example, imbeddiang experimental procedures 


in existing structures in as nondisruptive a mannar as possible, | 
replicating experiments, using placebo treatment groups, exasining 
the utility of proposed research, and not overgeneralizing fron 
results. Adult ‘educators also aust be concerned with the 
generalizability of their research. They must be applied researchers, 
eaphasizing external validity over internal.validity. (LMS) 
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Introduction 


$ . . © 3 


The purposes .of ‘educational research are to explain and predict 


an behavior. jt is usually unwieldly to undertake research about 


ny 


ople using everybody as subjects and everything‘as variables (although ° 


is has not stopped el from trying!) , If I may be permitted an 
4 aoaieay. educational researchers ‘iicate be anhietiod with biting off a 
miniscule piece of a delicious’ pie and attempting to comment of the 
hidden qualities of the visite pie having tasted only a small mouthful. 
“To pontdune the ‘anaipoy, most work in educational research has focused 
on two main aréas; a) how and from what part of the pie should the 
mouthful be taka’, and, b) what can you legitimately say about the 
“s 4 whole pie having ravigshed only a small saamla This paper addresses 


both of these important questions. : ' 


td 


ma 


~ 


-) Ae 4 ; ‘ ' 
' havé fallen prey. to{ the use of. techniques which may be of| use ‘in 
‘ : } ' qi 


eS . 
/* The ProblemA A Theoretical Perspective .. 
” Se 


tfaditional research literature on eoecens design and 


evaluation of Programs or processes ‘(campbell and stastev, 1963) € sting- 
- fits a cals two general types of threat to the. donerandzabilsty of 
experimental findings; ERE | and gxternal validity, mente |" 
oe \ 
validity (interpretability) is affected by actions such’ as Se vig’ 
@of statistical ieiiatii rs: or iabanstony design for ‘the purposeg o 


treatment. External validtty, which subsunes internal validity} is ‘- 


governed ‘by the broader! concept of yedlanil galt i.e., the epresen - 


/ 


ativendss or appropriateness of experimental findings when app ied to 
other Pepylastons Traditionally, the education researcher hag | focused 


on internal validity as ‘a means of inermandig overall validity/: Experimen 
C ven 
fave been made "tight", siiijacte and subject behavior have’ be¢n closely 


sontyatied. Unfortunately, the result has often been thé foy ation of 
{ es : 
"lab-wise" but "world-naive" principles. 
The problem with such research is. not that it is not hoa ‘It-is-- 
within the reebriched ‘parameters within which it ia, , MemaeTT The 
_ difficulty, comeg,_ (and it isa diffici lty that all researchers are .con- 


fronted with), whén lab or "tight" rebearch findings must pe used as 
1 < 


describers or predictors of real, non lahoratory .behavior 
r x By 5 
In the main, ‘relatively minor importance has been a tached to 


external validity in the, educational literature, In seeking ‘to emulate 
‘ . ‘ 
** a. ' 7 
their more "rigorous" older brothers in the fesearch game, educators _. 


D 
'~ 


differentiating between crop yields in sesponse, to fertilizer, but are. 


. 


external ‘validity in applied exper ntal dSsign. They deal with ts % is ae 


. f " 
different of external validity; 


pop’ ulation validity and eopiogiosi” : 


validity. Und population validity, \they deal with the inhere 


difficulties in generalizing from samp] s (usually student "volunteers" 


who, if they are thing else, are accessible), to target populations 


which often differ \ kedly ‘em these sence ; They offer the possibility : 


that the interaction\ ‘of es characteristics with experimental variables . \ 


a: @ ‘ \ 


\ 
(subject by variable contamination) is a ca Siuteacdon to the general- 


z « 


izability of current research eas Ecoldgical validity is character-:: 
ized by the eee inherent in vironment of the experiment; - 
description ‘of independent variables,’ multiple —— of' multiple 


one &, Hawthorne effetts, and errors due tes dependent. variable 


measurement, atc. , _ " ‘ ae es 
‘- Snow (1974) in carrying on: dhe sick of Hache ant Glass ‘(19687 
indicates that. the biggest piivenk to external validity. in all types of . ; 

; educational experiments edtans when the experinent: does not fit the 
nature: of the phenomena being axidiea, In short, he is concerned with 


. 


the imposition of experimental convenience, on real non-experimental 


phenomena. Snow (1974) er aera that a aeldatar problem develops 


“ye when’ no method is included in the Saroiu — to discover this” 


ss 


type ae ‘error factor. Area (1974) supports his ensay by aii the’ 


‘york of Pepnawick (1956) in which it was Heciawated that variabjes Co ge 


‘ +t . ry “e % t 


associated a time and space cannot be legitimately rey sided . 
. . . . s . 


., pendent. In other words, if one views that in some way the behavior 


ss ‘ 7 . . . ‘ e 
° 


gee  % 
i i = » F ¥ 2 “ry 2 
* ¢ 
* oes : . ea 4. % , , 
4 t 4 Rie oe é 
sl 2 of an individual is contingént on thé environment in which ha operates, P. 5 wi 
: re propdsition very difficult not to accept):, the description of that ' ;*. aa 
€ * . - 


subject's s*behavior in an alien ‘environment. (independent from the host’ 
‘ ‘ o. . ee ° 
. environment) - ‘and an attempt td generalize those laboratory ‘Eindinge to 


~ thie veal world situation, is folly. Many variables, which are, crntiiceis 
: ‘absociated in nature, ill'serve the process of educational’ research by 
. being separated arbitrarily in factorial. research design. 


rs 
», 


6 4 — 


' Given the agritulturdl foots of classical analysis of variance 


HRS a ® paced 


a pand factorial. design’ ‘onferinent, it is may to see how ‘the experimental. » »* 
ry 


Pi _— — ‘cone ‘to b tlewed asa penis object (with the i ; We _ 


or, educational’ program, ‘ioe active evaluated entity) . ‘Traditignal ian 


-exper iments are non-ecologically astad. tend to eanipileta ‘the, subject , r 
ae, e 

for r experimental. ‘convenience, and yield results which,” although “weight”, 

are non-representati e of the 2 wortd-at-large or the population of" the_ 


world-at-large. : ¢ ' - 


. : a , fg fie Ade age gene Ngee 
rene at”, ef ’ ee ai elle Seda tates ack 
3 aes \ 2 oa ° SF as . ¢ , me 

. . - 


i 2 4 Applied versus Basic Research | : a 
\.- «'% (a, ' uf 


Basic research reférs to hee ‘ zene activity. whose immediate aim is. 


the quantitative diovinsbeeti att vanttiaid grea laws; and whose’ 


ultimate aim is establishment of a system of concepts and relations 
(the so-called nomothetic net) iin which all specific propositions are | 


. deducible ‘from a few: general (iid Va antes research seeks eternal 
3 “verities. rts hallmark is the carefully designed ; and well controlled é. 
ee eer whose conclusions _are rigorously tested for statistical — 4:8? . 

~* shen Ferme (Ebel, 1967, p. 81) ” Historically, basic research has 


i "involved a search for internal validity,/ a search reflected by Campbell 


and Stanley's (1963). treatise. © @ 


.Applied résearch is concerned more with ai ec problem solving. 


The solution of current, relevant paoblens or issues, and he formulations ‘ 


on which thee are based are | Remporary (Ebel, 1967). ‘. ’ _ 


Of the basis of “thane: two ‘definitions, it vould seem that basic 


. 


; research is the crown prince of all research with applied research a poor = 


’ 


‘\country cousin. But, is such really the case? The honored position of ? 


x genera iz ty based on a high degree o ntern exper menta 
Nee lizability b a high a rf. a imental 


\gigour. The demise of applied research has rested on the apparent un- . e) 


whe ' 


(goncern of” the applied researcher with factors such as statistical’ 


i 
_ Peumptlone, criterion-predictor contamination, variable independence, 


‘i and data hmsae eter. : ’ . ‘ es Sg 8 


a i a . 
Aside from: the obvious differences in the two. at cee how far ‘ 


Boe 5 ke ‘ ay . » 


apart are ss none saat basic research within the perspective of generalr * 


“izabitity? Gahari(t benbt liey, it should ry noted, must bathe sole factor 
‘* A 


in judging all good research. Are the results napoesed in experiment A wos 


(be that a@ basic or applied experiment er program evaluation) generalizable 


‘ 
ve . $ os . : 


to an analgous natural situation? There is no 0 point in researching a 
i uly unique situation. Given the criteria of generalizability, og aaa . . ; J 


of educational research,’ be that research basic or- applied, faite i . 
: meet the mark. Basic research, while maintaining a high degree of ye 


F cms be ae 
‘ 


internal rigour, often sets up an alien environment which largely : 
Re aa . 


* nores typical oubiwak by environment interaction. tt is not sufficient 


to undertake- educational research the results of which are ne - 
‘| 


» 


- 


c 


way as to preclude generalization. It is not so much that the bulk of - 


appligd r research is "soft"--it is Hete- ue rather that its objectives 


: | ; . 7 
ta any oe . . ; 
solely to other experi ents. ‘Conversely, applied research often attempts. 
to answer all questions, for all time, for all people. By attemptifg * 
‘ \ ef 3 
lobal whitewash, approach to subject selection, criteria,” and, variable 


“selection, the subject and the results are ‘often described ‘in such a 4 


, iS v4 5 
are ill defined, diffuse, and often’ unspecified. ‘ ve 
: . 


:° a fundamental diaries principle iia must govern generalizability- U 


in edticational design is representativeness. ‘ey this evitsries basic 


° 
2 


{research has failed, since most ‘experimental apie represent only other - 


experimental designs, and applied research has failed in its ‘ubiquitous 
. , z ¢ 
description of the interaction betweeg subject and environmental character- mp 


istics. In short, most basic resear fails miserably when viewed within 


the perspective of représentativeness nd quite probably ey ici ‘ 


seme “research -dees TO- -better (with~ most applié research programs; so little J 


” 


"generalization are additive. Ng 


information is aria ehae it is not possib¥e to rake guch a qualitative 


decision). Generalization from minced eines studies is largely a 


. 4 Ms AY . es 
two-step procedure. In “step one, we’ quieraids -from the sample of subjects 


° i \ 7 
involved in the experiment to the restricted nial populgtion from s ~ 


a3 6 experimental a was drawn. In ste two, and it bess here, that 


most ae are.on any crenet, we generalize fr Kom the accéssible 


. 


population (students, program participants, etc.) a larger population 


with which one is ultimately concerned. Errors@in making’ this twe-step Be a 


. 4 


* 
= @ 


_ randomness. Such randomness, however, es not ext Sone by. It is ; 
difficult to -achievé in stép one;, it is impossible i achieve’ in step - a 9 I: 
two. ‘People, unlike the kernels ae wheat so at home with such techniques, _— 
‘do not eepatnementadtinn 80 easy. "8 ; 7 . ar _ 
Several researchers (Bracht “and Glass, 1968; Snow, 1974) have 
addressed this problem of twp-step generalization. Their findipgs 


"indicate that, although one cannot often generalize to the intended 


population ‘with any degree 'of certainty, it as possible ‘to generalize. 


‘ie 
to a “hypothetical accessible population like those involveg (Snow, 1974, : . “1 
Pp. 270) ." The: crux of this sonsbaliantlad, whether it be doné in steps * 4 

, ese or two, ie a aseeugty description of subject and environmental | : 

; chocactenieetes. Generatization then,’ is attempted cautiously, based..« _ “ee ‘ 

‘i x ‘ 
a, 4 


on a thoraugh evaluation of the similarity of characteristics between . i 
foe ; Te ae vo 


the two or more groups. Clearly, this is the job of/ ar educational 


im ‘ 
2 


craftsman. : 


i ‘ = 
3 Interactions of ‘sibjects Gbatacteiecice, as would be resaiay 


te Pa 


. hoted in the xeal world, are “also, an imprientnquaddacwtion: from the . : : % 


¥, . 
perandoliew of limitations to generalizability. A thorough ‘description 


“of subject characteristics--at both levels of smear tae Eames 9 


. 


the ar aa of. such interactions. ‘Snow 162974) notes this Reine’ in 


‘ 


discussing SRECtent generality (the vange of cheracteristias investigated r 
’ . a 


in ary ae experiment) a ' Design, he argues, ad atrengtheded’ by increas- 


site the "rarige 9 traits under consideration and the Sriterserdors between’ 


. 


those characteristics. These points nie. echoed in an important paper ‘by : ass 
Clark and Snow (1975) im which they caveat six alternative designs for 
instructional technology recon. all. of which require increased speci figity 


‘ 


in subject description as a precirsor to uteisihe fequeseipuniike ona | 
c" 


ae i Meee ; —e. 
?, . ° 
ve etl + , . : 
- wy / Ze \ va 8 Pe 
a : sv ‘ J 5 : % : “, > . nm . . 
ee. i ri o. : / 
8 ayes - Strate Co ras fo Educational ‘Researchers - wd 
ae Strategy Cohcerns for Educational Researchers 
@ 
. x 
, 7 What course is ‘open to ane educational researcher in | general and ‘ 
. i the continuing education ‘researcher specifically if he or _— is to build 
a re an ar * ¢ Os . 


significant | esearch studies based ona consideration: of the Limitations 
| 


oP to’ generale: ility inherént in many erbaiietonal designs?. As outiined 


: below ’ chat 


ea 1 number of Significant considerations: a 
2 r) ) s r 


1. g-* If one is to truly’ satiate a Program, of : 
sit - : J ee eee cbincern muSt be’ the eed nkeirtiny of the reality of thet - 
ire : ee duritig’ the panes of e research study. To the extent. cs 

an : ‘the research study itself ane s the'fundamental natyre of the program , 
w 


. ‘being evaluated, the research findings from Sab program are non- 
. - e, 
7 ; , ira liebe, Experimenta] pienaluves need to be imbedded in , 


Yo ca \ 
te@xisfing structures in as non-disruptive a manner as possible~ 
2. . Experimental Replication. The branching out of the program being 


4. . . a 
ue evaluated (e.g., systematic replication in time or consequence) 
. ’ wi . . ry ; 9 
provides one index, of possible generalizability. Most researchers 


“know that oe given. point, ng matter — eiliyy can be supported “by 
: > 
oy the research.” Replication sagt use a number. of questionable aN 


wee ny (3 . 


{ 
. ideas to “die. in ‘the FR rn is needed in educational seicriesii ‘is 


° 


not brasking of ‘new cnet but the ‘myetenatic inves €1gation of: what 


\ has already been seeded. Glass (1976) naiebe this point well in his . 
’ 7 ry ? 4 
? ¢ P . é : - - 
: aan description of meta-analysis (andlysis of analyses) as a necessary | 
‘ Joe ” ; , . 
ed . and important form of ‘résearch. L : oe a 


© 3. Internal\Experimental Investigation. Most aia ag jin describing an 


experiment, a researcher will describe only | subject behavior 
ie: . Bn comparison to entering subjéct behavior, Process variables are u 
cy ° N . ; 
e \ 
often ignored. Internal documentation and observation of subject 


3 ‘ ‘ ’ . . oie Ss 
, 1 a a 
‘ a 4 . : 


“g-. 


£ 2 oan a os 
” . 7 é . . 7 8 & 
-- . : . ‘ 
. é e . 
° , . a 
. 


‘behaviors (par, icularly im rtaht’ in program evaluation) can d 
34 P- 


/ 
a 
en to explain’ confliéting pee ‘called’ _ naan results." 


sot External Esperisentil Investigation. Do all experimental subjects 


: paceepe orlce a research abviy has been “completed? Do* ali” pencastonal: 


prograns cease operation on the publication of the research paper? 


« 


‘Such questions are unanswerable in a conventional reseaxch design. 


‘ 


The increase in information due to post-experimental investigation 


’ 
sy 


‘would undouptedly” lead sto an increase in generalizability due to an 


. v7 


F | ino in referent generality. rots : 


Subject Description. As nofed earlier, a thovoiiah description at 


‘ stbject characteristics, including aptitude variables and inter- 


4 ae ia? § ‘ Oy. 
action effects, is paramount if an increase: in generalizability is 


to. be achieved. It is important to know why things work,as well as ° 
% . " 4 af e . 


that they work. A binary (effective vempus ngn-ef fective) approach. 


; ! y. 
“to educational research ill serves.the process of education. 


« 


’ “os _ ‘ = 


, Systematic Treatment Description... Frequently, the literature reports 


educational studies,’ particularly program evaluations, where very ~- 


* . 


little, other than the:effects of the programs, is described. Thus, 


we see participants sulsjemkaa to a "Life Skills. Program", ai 


Progran" , "Community Planning Workshop" " etc., with little told cae 
the. reader about iia ee ae or phocess of such a program. Often, 


-yheg the. ‘reader attempts to duplicate such a program, aie results. 


are, disheartening. As has already been indicated, the dettainictioes 


° ’ ° 


effects of a eee by: Dasiniing progzam by learning environment na 
/ 4 2 fe ae ry ~ 4 
stats is probably mn most eure consideration in program 


- evaluation. ° However, interactive effects are largely igooxed in a 


oN - re ‘ .. e ae Ss 


simplistic evaluation of main effects, an ‘evaluation which hides as + 


much as it shows. “the mere réporting of program labels, without a 


thorough description. of the interactive process of such programming, 


is wholly ‘inadequate. - : 


Pl'acebo/Treatment Groups. Most fréquently, programs and/or ~-. 


4.) 3 


‘ ‘ : 
educational experiments are evaluated in terms of a no-treatment_ 


aN 


control group. What do the results of such evaluations really tell. 

us? The xesults’ could probably be summarized as "something is better 
A , 

than nothing." The truly important question in relation to program - 


evaluation is their efficacy vis a vis existing and/or alternate 
¢ : = . 


_ programs. This is. not to say that the use of no-treatment control 


. . 


groups is wrong--it is. simply inadequate. . y 


Nipping in the Bud. Stated bluntly, ‘there is a great deal of 


research that is not worth doing; it is simp waste of time..- - 


It is not wutbtotant to "do the best = ‘can within the constraints 


a 
wiven. -us."" If proper, attention cannot. be paid to impottant-* ° 1 


Pd) 
WO weer oo & - 


research considerations, the sued dught ndt 0. of bothered with 


from a research perspective. The problem with research is,that, 
1: : 


oak 


once completed, pearly defined and operationalized studies .often 


nw ay 


receive the same or’ more weight as well conceived and p Seerenseneliaee 
ones.’ Good research abtece in this "lowest contmon denoninatar” 
approach.. universities. parbicilertly are to blame in this area ain 
coercing ‘their graduate atudanks to "go through the motions" in. 


s Sf " : . 
undertaking research that is both unimportant and uninteresting. 


. . . 


ree More attention needs Yo tie paid to the utility ast pc research . 
S : Y 
rather than atempeisy to patch-up a sick study vBacn should have’ 


we # 
oe aborted on conception. The difficulty may be that we often 
_ look aia a Study, . dommi t - ourselves to it in an evaiugtion sense, - 
: 4 “ . . z 


‘he 4 and cid afte to do the best possible within. the constraints 


eo @~ _ given. aaa on in progral evaluation; basic go-no. go Secieions te tate 4 
at? * ee to be ae That is not to say that the program itself | 

: * phontTA nek ga on, but it should not be icrviniawnty: prea 4 oat 

il *y, sf pet ey “associated with an: evaluation process., r% “oe = ee ¢ 

t % oN ‘Top Much of ad sis Thing. The urge to irgeniailen 1a a frequent ? 

‘ clandaton e educational ainitaaieaada A professor’ that Ioonce . _* - : 


’ ». 
. " i know many years ago nade siisate he “thought was a world shattering . “PRans 


. . . * 


‘ 7 observation. He said that the. human organism seems largely incapable —— 
saa. : of carrying in consciousness more than tvb alternate hypothesis. 
Thus we have hot-cold, good- : ofsiclt-inefticsent, .sick-well, 
: etc. The result is ‘Bak we pe take: the findings from a very : 
restricted study and overgeneralize to populations that are eer 
7 dissimilar. Too much "is often madg of too little. , * 


“a 
* « 


_ The Continuing bducutar and the ak Dilemma _ 
f . The continuing educator is ie*eseiy the same position with ‘respect : 
to research programming as his more traditional ehonitor cousin. Needs — 
, research, program edlcabion: community. studies, sdlbeeiet educational | i | 
strategies, all are taken as important, albeit crucial aspects of the 


continuing educator's role. Within the framework adopted by Ebel (1967), ° 


the-continuing educator must be an applied researcher; he is concerned 


12. 


with the bolution pf immediate, problems, ones largely ephemeral in nature. 
Educational programming, however, does not occur in a vacuum. sie aa ilar 
aré seldom wholly tnkeiion “The ‘continuing educator is , tust be, concerned 


. es with the betarutieiitines of oa findings, sa area. Ebel- + (1967) resetves,, ey P 


. at, ‘ ‘ ¢ ; 3 Sy, S.* » 
for basic research.. ae . oR. Re Hy 5 soos is 
However, pie criteria of siacavinl Sti nities é experimental ” ; ne. 


- a 


findings based on ite ‘representativeness of the research undertaken,” ° 


s 


@istinctions between basic and applied research become less clear. For, 
: i ® 


if any research is to be good, it must be generalizable. To be generalizable 
is toginclude elements of both of the traditional concepts of applied oe 
a basic nepeaech strategies. Simply to’ undertake research in order. to 


Seatinade & single program is insufficient, if that research cannot be 


” legitimately applied to similar programs. However, to do reseavch on , _ 
¢ 4 
~ . prograsming conceptually is also insufficient since applicability is : a 
largely ignored. The continuing educator/research must be concerned with 1 8 * 


the ptoblems inherent in the specific program being qvalupted, but he must” 
algo be’aware of what Ebel’ (1967) calls the rere ee net (p. 81)." 4 

* oo some that will say that the approach I ae proposed 
in this paper, Which emphasizes an external es than an sanesioat : 


. 
- . é .s 


experimental orientation to research, is antii-scientific, even anti- : ¢ 
- ' : . 


° 


education. tet me assure “you that such is t oe case. I ama . 
thorough-going mabtemelptogioal se a r atin helieve: in the 

° * e 
"magic" of research and the- Sictie of idvvuléiosiott syotens based on 


other than rhetoric. Where a differ teem thany of a colleagues an the * 


* 
research some ia a belie§ that in emulating our "haga" acience brothers, s 


ie . 


%. : + oF — = 13. 


. « zs 
we are seeking to catch the wind. To be an educational researcher is fo 
Live with confusion, ‘but it is also to seek to dispel a portion of that 


-s . 


confysion through a scientific attitude of inquiry suited to our 


discipline. The hope, the oniy real hope as I see it for ehicationnt 
ra ninendhs is contained in the notions of generalizability and represent- 
- ativéness as I have attempted to outline them in this brief: paper. 

7 To continue the enatogy posed at the start of this paper, it 
* may be not so much that we have been overly careful in selecting the 
piece of- pie we wish to taste but that we ‘have wither been reluctant to 
re taste the pie at all or that we say toq muck about the miniscule 

piece we have tasted. . oF. 
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